Penalized maximum likelihood for multivariate Gaussian mixture

In this paper, we first consider the parameter estimation of a multivariate random process distribution using multivariate Gaussian mixture law. The labels of the mixture are allowed to have a general probability law which gives the possibility to modelize a temporal structure of the process under study. We generalize the case of univariate Gaussian mixture in [1] to show that the likelihood is unbounded and goes to infinity when one of the covariance matrices approaches the boundary of singularity of the non negative definite matrices set. We characterize the parameter set of these singularities. As a solution to this degeneracy problem, we show that the penalization of the likelihood by an Inverse Wishart prior on covariance matrices results to a penalized or maximum a posteriori criterion which is bounded. Then, the existence of positive definite matrices optimizing this criterion can be guaranteed. We also show that with a modified EM procedure or with a Bayesian sampling scheme, we can constrain covariance matrices to belong to a particular subclass of covariance matrices. Finally, we study degeneracies in the source separation problem where the characterization of parameter singularity set is more complex. We show, however, that Inverse Wishart prior on covariance matrices eliminates the degeneracies in this case too. INTRODUCTION We consider a double stochastic process: • A discrete process (zt)t=1..T , with zt taking its values in the discrete set Z = {1..K}. • A continuous process (st)t=1..T which is white conditionally to the first process (zt)t=1..T and following a distribution: p(s|z) = f(s;ζz) In the following, without loss of generality of the considered model, we restrict the function f(.) to be a Gaussian: f(.|z) = N (μz,Rz). This double process is called in literature "Mixture model". When the hidden process z1..T is white, we have an i.i.d mixture model: p(s) = ∑ z pzN (μz,Rz) and when z1..T is Markovian, the model is called HMM (Hidden Markov Model). For application of these two models see [2] and [3]. Mixture models present an interesting alternative to non parametric modeling. By increasing the number of mixture components, we are able to approximate any probability density and the time dependence structure of the hidden process z1..T allows to take into account a correlation structure of the resulting process. In the following, for clarity of demonstrations, we assume an i.i.d. mixture model. CHARACTERIZATION OF LIKELIHOOD DEGENERACY We consider T observations (st)t=1..T of a random n-vector following a multivariate Gaussian mixture law: